Modem mass storage systems continue to provide increasing storage capacities to meet user demands from host computer system applications. A growing reliance on large capacity mass storage has fueled a corresponding demand for enhanced reliability of such storage systems. One popular solution to the demands for increased storage capacity and reliability is the use of multiple smaller storage modules configured in geometries that permit redundancy of stored data to assure data integrity in case of various failures.
RAID (redundant array of independent disks) disk arrays are an example of a fault tolerant, mass storage technology that has developed in response to the ever-increasing demands for greater storage capacity and reliability. RAID disk arrays supply host computer systems with large amounts of storage capacity in addition to providing redundancy of stored data to assure data integrity in case of various failures. Such disk arrays therefore typically include redundant components such as controllers and power supplies, as well as hot-swap capabilities for various subsystem modules (i.e., an ability to change-out modules without powering down the system).
Conventional RAID arrays commonly have two controllers that manage the array and perform mirrored memory operations for data redundancy. The controllers make the array appear to the host computer as a single, highly reliable, high capacity disk drive. Both controllers have independent access to all data cache information, all input/output (I/O) state information, and all system state information so that a failure of one of the controllers does not prevent the remaining working controller from accessing all the necessary information to take over sole operation of the array. Significant bandwidth is required on controller interconnect buses to allow the controllers to transfer the necessary information for processing host I/O requests and performing mirrored memory operations.
As disk arrays become larger, controller pairs can be added to the arrays to increase their computing resources and maintain or improve system performance. However, as the number of controller pairs increases, the amount of data flowing between controllers over the controller interconnect buses increases dramatically. As an example, when a controller pair “A” receives a host computer write command that is destined for the cache memory on controller pair “B”, the controller interconnect not only carries mirrored data traffic and inter-processor communications between controllers in pair “B”, but it also carries pair-to-pair traffic between the two controller pairs “A” and “B”. First, the interconnect must carry the traffic from the controller board in pair “A” that received the host data to the controller board in pair “B” that is the destination of the host data. Second, the interconnect must carry the mirror traffic between the two controller boards that form controller pair “B”. Therefore, an increase in the number of controller pairs within a disk array can contribute to performance bottlenecks due to bandwidth limitations of the controller interconnect buses.
Another consideration regarding controller interconnects is emerging technologies that allow for wider interfaces between disk arrays and host systems. As higher performance host computer connections are developed for connecting RAID storage arrays to host computer systems, controller interconnect buses experience a corresponding increase in the amount of data flowing between controllers within an array. Again, bandwidth limitations on controller interconnect buses within the array can result in performance bottlenecks.
Another problem that results from adding more controllers to a disk array (i.e. clustering the controllers) is that more and more data travels to remote controllers rather than a local controller in the mirrored controller pair where the data is received. Where there are only 2 controller boards in a disk array, all the host computer disk traffic is destined for the local mirrored cache because there is only one mirrored cache. However, when there are 4 controller boards in an array, the percentage of data flowing to the local mirrored cache drops to 50%. Half the traffic stays with the local cache memory while the other half is destined for the remote pair's cache memory. With 16 pairs of controllers, only about 7% of the traffic is local. Thus, the characteristics of the controller interconnect changes dramatically with the clustering of controllers in the disk array.
Another important consideration regarding communications between controllers in a clustered disk array is the effect that failures in the controller interconnect have on the operability of the array. Currently, a failure in a controller interconnect can result in a failure in the operation of related array elements. In order to avoid a permanent lock-up of the disk array under such circumstances, various timeout functions must be designed and built into the array hardware. This causes difficulties in hardware design and also increases the complexity of firmware that must be able to tolerate the loss of controller communications without notice.
Accordingly, the need exists for a controller interconnect structure in disk arrays having clustered controllers that provides for the efficient use of current and future interconnect bandwidth capabilities and that enables continued controller-to-controller communications and disk array operability under various interconnect failure scenarios.